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SUBSTITUTE SPECIFICATION 

[0001] METHOD AND APPARATUS FOR SIGNAL PROCESSING 

USING RECONSTRUCTION FILTER, COMPUTER PROGRAM 
PRODUCT, COMPUTING SYSTEM AND CAMERA 

[0002] BACKGROUND 

[0003] Digital cameras based on digital signal image sensoring of e.g. video 
and still images may be advantageously equipped with an image sensor comprising 
a red-green-blue (RGB) Bayer-color-filter-array. In such RGB Bayer-color-filter 
array each pixel senses a red, green or blue primary color in a predefined pattern. 
This pattern is built up of alternating green/red columns and green/blue columns. 
Such a sensor may have a limited resolution in comparison with a camera using a 
separate image sensor for each primary color. However, a camera with three image 
sensors has three times as much pixels contributing to the resolution than a single 
RGB Bayer-sensor. Using three sensors is for most applications disadvantageous 
due to cost and size requirements of the application. On the other hand, when using 
one single image sensor to sense all three primary colors red, green and blue within 
one single array, advantageously in a RGB Bayer-color-filter array, it is necessary 
to reconstruct missing pixels of certain colors to process a consistent whole of a 
picture. Due to the RGB Bayer-structure, different Nyquist-domains respectively 
with regard to green, red and blue colors result in a color dependent resolution and 
possibly aliasing patterns. Nevertheless the RGB Bayer-structure is one of the best 
performing signal color arrays. 

[0004] Several interpolation schemes may be provided to increase signal 
quahty. In WO99/39504 a conventional method of interpolation is describSed in 
rather general terms wherein an intermediate color signal is interpolated at 
positions where no signal of a given color is present and an average of a given color 
is generated. 
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[0005] A further method for signal processing uses a more advantageous 
interpolation scheme as described in the WO99/04555 and in the European patent 
application with the application number EP 01 200 422.2 not yet published. 
However such methods still suffer from e.g. aliasing of a free luminance signal or 
further signal distortion. Such signal distortion especially results in an erroneous 
generation of false colors in an image. 

[0006] In WO 99/04555 a green reconstruction method a for RGB Bayer image 
sensors has been described merely concerns the green color reconstruction. The red 
and blue colors remain reconstructed in a conventional way. Merely a missing green 
pixel is reconstructed. The reconstruction of the missing green pixel is carried out 
by means of a median filter sorting three specific variables: two of them are derived 
from the green color, the third one from the red or blue color. A disadvantage of this 
method is, that for high saturated colored edges artifacts are introduced which look 
like the border of a postage stamp. The algorithm disclosed in WO 99/04555 will be 
referred to as the smartgreenl-algorithm. The smartgreenl-algorithm is based on 
the concept that resolution losses are best observed at high frequencies near white 
scene parts and less good near colored parts. Keeping this in mind, the contribution 
of the red and blue pixels is used to help determining the reconstruction value of the 
missing green pixels. The object of the smartgreenl-reconstruction is to maximize 
the resolution of the green color. For this purpose the median filter algorithm is 
applied as follows: naturally, the location occupied by a red (R) or blue (B) pixel is a 
location of a missing green pixel. In the smartgreenl-reconstruction-algorithm a 
center value of a 3.times.3 pixel array, also called the median value, is applied for 
the reconstruction of the missing green pixels. Consequently, the simple median 
filter for green merely replaces a conventional interpolation concept of green 
reconstruction, whereas the conventional red and blue reconstruction method is 
maint£dned to be a simple interpolation. Luminance filtering, color filtering and 
contour filtering is also restricted to the filtering for green. False color detection is 
merely based on a conventional interpolation concept of green reconstruction, and 
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also the conventional red and blue reconstruction is maintained to be a simple 
interpolation. 

[0007] Such smartgreenl-reconstruction method improves the resolution of 
the green pixels in the horizontal and vertical direction with the aid of the 
information of a red and/or blue pixel. This conventional method relies on 
interpolating the color sample to be interpolated in dependence upon neighboring 
color samples of the same color and a differently colored sample merely from the 
same location. As a consequence, the reconstructed signal suffers from red- and/or 
blue-colored aliasing. Vertical and horizontal colored edges suffer from a green 
intensity modulation in the respective direction resembling the border of a postage 
stamp. 

[0008] Further improvement as outlined in EP 01200422.2, herein referred to 
as the smartgreen2-reconstruction-algorithm, was able to significantly improve 
resolution but not to remove the mentioned disadvantages of signal distortion and 
signal aliasing. In particular at edges and high frequencies some signal distortion is 
still visible such as alternating colors with neighboring pixels. Artifact black and 
also white dots are generated erroneously. 

[0009] Parallel processing of a color- and a contour-reconstruction filter in 

principle is known from WO99/04554. However, this conventional concept suffers 
from the same disadvantageous as outlined above. 

[0010] SUIMMARY 

[0011] The invention regards a method for signal processing, wherein a sensor 
signal of an image sensor is provided as an input and wherein the input is 
reconstructed in a filter to establish an output for further processing, wherein the 
filter comprises at least one reconstruction-filter selected from the group consisting 
of: a luminance-reconstruction-filter, a red-green-blue-color-reconstruction-filter 
and a contour-reconstruction-filter, wherein the input comprises a plurality of pixels 
and a pixel provides a color value assigned to at least one of the colors red, green or 
blue. The invention also regards an apparatus for signal processing which is in 
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particular adapted to execute the method; comprising an image sensor for providing 
a sensor signal as an input and a filter for reconstructing the input to establish an 
output for further processing, wherein the filter comprises at least one 
reconstruction-filter selected from the group consisting of: a luminance- 
reconstruction-filter, a red-green-blue-color-reconstruction-filter and a contour- 
reconstruction-filter, wherein the input comprises a plurality of pixels and a pixel 
provides a color value assigned to at least one of the colors red, green or blue. 
Further the invention regards a computer program product, a computing system 
and a camera adapted for signal processing. 

[0012] BRIEF DESCRIPTION OF THE DRAWINGS 

[0013] The Figures of the drawing show preferred embodiments of the 

invention and are enumerated as follows: 

[0014] FIG. 1: Location of RGB- and contour-reconstruction in a memory 
based architecture; 

[0015] FIG. 2: Basic block diagram of the smartgreen3 reconstruction 
(particular information to the block regarding the multiplication of the R- and B- 
colors with the smartgreen parameters may be taken from WO 99/04555); 
[0016] FIG. 3: Realization of the white compensated luminance pixel Yn; 

[0017] FIG. 4: Several white compensated luminance reconstruction filters; 
[0018] (FIGS. 3 and 4 illustrate the white-compensated-luminance signal of a 
preferred embodiment which is advEintageously fi-ee of green-green differences. 
Green-green differences may be removed by a restoration of a green uniformity of a 
Bayer image which allows to eliminate green-green differences in the green signal 
of the sensor with maintenance of a Laplacian (i.e. smartgreen) RGB-reconstruction 
method and without visible resolution losses. Also they may be removed by 
preventing green non-uniformity in the parallel contour signal of RGB Bayer image 
sensors, which may be performed by developing a two-dimensional parallel contour 
filter that eliminates the green-green differences caused by the image sensor.) 
[0019] FIG. 5: Three 4.times.4 contour filters; 
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[0020] FIG. 6: Three 6.times.6 contour filters; 

[0021] FIG. 7: Furtlier simplification of the smartgreenS block diagram of 
FIG. 2 by using one contour filter; 

[0022] FIG. 8: Transfer characteristics of the three 4.times.4 contour filters of 

FIG. 5; 

[0023] FIG. 9: Transfer characteristics of the three 6.times.6 contour filters of 
FIG. 6; 

[0024] (FIGS. 5 to 9 illustrate a preferred embodiment capable to provide 

aliasing free parallel contour signals. This may be achieved by either parallel color- 
and contour processing for Bayer image sensors, which allows to use the green 
signal from the image sensor for generating a two-dimensional contour signal in 
parallel with the RGB color reconstruction. The advantage is that no extra row 
delays are needed as it is the case with serial contour processing. An aliasing free 
contour processing for a RGB Bayer image sensor, which is based on a unique 
5.times.5 parallel contour filter that, has no need for an optical low pass filter and 
has a zero throughput at the first RGB sample frequency. Its signal distortion is 
almost zero, resulting in a contour signed without visible artifacts. The so far laiown 
contour filters amplify the back folded and undesired frequencies outside the 
Nyquist domain of the sensor as well. This will lead to distortions and as a 
consequence a better visibility of the unwanted aliasing components in the picture. 
This unique 5.times.5 filter prevents those aliasing artifacts and moreover 
eliminates the green-green differences caused in the green channel of the image 
sensor.) 

[0025] FIG. 10: 3.times.3 green reconstruction for center green is present; 
[0026] FIG. 11: 3.times.3 green reconstruction transfer for center green is 
absent; 

[0027] FIGS. 10 and 11 illustrate a preferred embodiment; 
[0028] FIG. 12: Basic block diagram of the smartgreen4-reconstruction; 

[0029] FIG. 13: Details of smartgreen4 showing 3.times.3 RGB reconstruction 

in combination with 5.times.5 aliasing free contour reconstruction; 
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[0030] FIG. 14: Top-left: S.times.S RGB reconstruction with a 5.times.5 

aliasing free contour filter, top-right: an aliasing fi'ee contour-filter only, bottom left: 
a 3.times.3 RGB reconstruction with a 5.times.3 contour-filter, bottom-right: a 
5.times.5 contour-filter only; 

[0031] FIGS. 12 to 14 illustrate a preferred embodiment i.e. a smartgreen4- 
method with 3.times.3 and 5.times.5 filter weights of smartgreen3 as shown in 
FIGS. 10 and 11, wherein there is no need for a 2.times.2 post filter. 

[0032] DETAILED DESCRIPTION OF THE DRAWINGS 
[0033] This is where the invention comes in, the object of which is to specify a 
method and apparatus for signal processing, and also a computer program product 
for signal processing, a computing system and a camera adapted for signal 
processing such that signal quality is improved. In particular, a signal should be 
improved with regard to signal distortion, and aliasing, but the signal should still 
provide sufficient resolution. 

[0034] As regards, the method the object is achieved by a method as 
mentioned in the introduction wherein in accordance with the invention the method 

further comprises the steps of: 

[0035] applying the luminance-reconstruction-filter to an array of pixels of 

predetermined array size comprising a number of pixels, wherein at least one of the 
number of pixels is formed by a red-pixel assigned to the color of red, at least one of 
the number of pixels is formed by a blue-pixel assigned to the color of blue, and at 
least one of the number of pixels is formed by a green-pixel assigned to the color of 
green and weightening the red- and/or the blue-pixel by a green-parameter, and 
summarizing the pixels of the array into one output-pixel, and entering the output- 
pixel in the array, and applying the contour-reconstruction-filter in parallel with 
the color-reconstruction-filter. 

[0036] As regards the apparatus the object is achieved by an apparatus as 
mentioned in the introduction wherein in accordance with the invention it is 
proposed that the reconstruction-filter is adapted to be applied to an array of pixels 
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of predetermined array size comprising a number of pixels, wherein at least one of 
the number of pixels is formed by a red-pixel assigned to the color of red, at least 
one of the number of pixels is formed by a blue-pixel assigned to the color of blue, at 
least one of the number of pixels is formed by a green-pixel assigned to the color of 
green and the apparatus is further comprising: means for weightening the red- 
and/or the blue-pixel with a green-parameter, means for summarizing the pixels of 
the array into one output-pixel, and/or means for centering the output-pixel in the 
array, and means for parallel processing of contour- and color-reconstruction. 
[0037] The term pixel herein is used in particular to refer to the value of a 
color sample in the signal. 

[0038] The present invention has arisen from the idea to provide a concept of 
flexible design for the reconstruction of images. The main idea is to provide concept 
of reconstruction based on a white compensated luminance-reconstruction. This 
idea basically is realized by parallel processing of color-reconstruction and contour- 
reconstruction. Weightening the red- and/or blue-pixel with a green-parameter is a 
further major improvement, whereas prior art concepts merely rely on the 
reconstruction of a missing green-pixel. The proposed reconstruction-filters are 
designed to be applied to an array of pixels of predetermined array size. 
[0039] Consequently filtering is performed in an advantageous way on this 
array. Whereas conventioneJ methods rely on simple interpolation in dependence 
upon neighboring samples or a sample of the same location, the proposed concept 
provides specifically adapted reconstruction-filters which take into account all 
pixels of the array. 

[0040] Parallel contour processing for Bayer image sensors allows to use the 
green signal from the image sensor for generating a two-dimensional contour-signal 
in parallel with the RGB-color-reconstruction. Advantageously no extra row delays 
are needed as it is the case with serial contour processing. Further advantages will 
be outlined in chapter 2 of the detailed description. 

[0041] In a further configuration advantageously a contour-reconstruction- 

filter is applied, in particular parallel to the luminance-reconstruction-filter and 
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their reconstruction signals are added. The contour-reconstruction-filter 
advantageously is applied to an array-size of 5.times.5, in particular to an array- 
size of 4.times.4 or 6.times.6. By the proposed method advantageously the transfers 
of the red, green and blue reconstruction-filters are matched in such a way that less 
or no greenish interferences will occur at higher diagonal frequencies as it is the 
case with conventional filters. The filters are free of post-stamp-border looking 
artifacts and free of color- and luminance-edges. Further all generated luminance 
signals are free of aliasing and distortion by means of the so CEdled luminance- 
white-compensation in combination with the filter weights. 

[0042] The proposed method offers the possibility to apply several aliasing 
free contour-filters which may be chosen as a function of the optical transfer of the 
camera. In particular an array size to which the contour-reconstruction-filter is 
applied to exceeds the size of an array a color-construction-filter is applied to. 
Advantageously the contour-reconstruction-filter is applied to an array size of 
4.times.4 or 6.times.6. 

[0043] In a further preferred configuration the reconstruction-filter is a 
luminance-reconstruction-filter and the pixels of the array are added together in 
one white pixel being the output-pixel. ]VIost preferred the green-parameter or a 
number of green parameters are chosen in dependence of a sensor matrix of the 
image sensor. IMoreover the green parameter or a number thereof may be chosen in 
dependence of an optical transfer of an optical system providing an image signal to 
the image sensor. Thereby the RGB-color-signals are advantageously reconstructed 
with filter weights that can be chosen as a function of an optical transfer of a 
camera in combination with the heaviness of the sensor matrix. Most preferred two 
green parameters are provided. 

[0044] The basic concept of luminance reconstruction as described above 
implying with one or more of the preferred configurations is in the following 
referred to as the "white compensated luminance-reconstruction" for RGB-Bayer- 
image-sensors, or simply "RGB-reconstruction". The green-parameters are also 
referred to as "smartgreen-parameters". Using the smartgreen-parameters also 
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with a red- and/or blue-pixel as defined by the proposed method and the further 
developed configurations thereof will be referred to as the "smartgreen3"- 
reconstruction method. Particular ways of determining the green-parameters are 
also described in WO 99/04555 and EP 01200422.2 and may be appHed and used 
within smartgreenS as well. 

[0045] The specific kind of arranging filters and filtersize of the proposed 
method results in £in aliasing fi*ee signal and in particular also free from green- 
green differences. Details will be further outlined in chapters 2 and 3 of the detailed 
description with regard to preferred embodiment and with reference to the drawing. 
[0046] The proposed white compensated concept advantageously results in an 
aliasing free luminance signal, even at the multiples of the sample fi'equency in 
case of a camera without optical low pass filter. IMoreover this white compensated 
luminance signal is free of signal distortion. The basic method and apparatus as 
proposed are adapted to offer a broad and flexible extension. It is possible to offer a 
variety of several reconstruction-filters which may be chosen and adjusted e.g. in 
dependence of an opticed transfer of an optical system and/or a sensor matrix of an 
image sensor. The proposed method and apparatus are capable to maintain a rather 
independency of optical low pass filters. This is particular advantageous as camera 
designs of potential customers may vary. The proposed method and apparatus are 
capable to implement adjustable false-color-filters in various ways and in a simple 
manner as will be outlined below. 

[0047] Continued developed configurations of the method are described in the 
dependent method claims. The proposed apparatus may be improved with 
respective means for executing the method. 

[0048] In particular it is preferred that a center-output-pixel of a second filter 
subsequent to a first filter is positioned in phase with the output-pixel, in particular 
the center-output-pixel is centered at the same center position of the array as the 
output-pixel. IMost will be further described below. 

[0049] Specifically a luminance-reconstruction-filter is applied to pixels of an 

array having an array size of 2.times.2 or 4.times.4 or 6.times.6 or larger if 
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preferred. In a particular preferred configuration the luminance-reconstruction- 
filter is applied to an array size of 2.times,2 or 4.times.4. The filter size may be 
chosen as a function of the optical transfer. Also the weights for the respective filter 

may be chosen differently. 

[0050] Additionally a low pass luminance signal may advantageously be 
generated by low-pass-filters applied to an array size of respectively 4.times.4 or 
6.times.6. In an advantageous configuration the 4.times.4 or 6.times.6 low pass 
filter is combined with the 2.times.2 or 4.times.4 luminance-reconstruction filter 
respectively to establish one single filter. None of the resulting signals suffers from 
green non-uniformity artifacts caused by the sensor. Further details of the 
luminance-reconstruction-filter may be taken from the detailed description and 
from a patent application with internal file number ID606638-I, which has been 
filed on the same day as this application and which is incorporated by reference 
herein. 

[0051] Nevertheless, depending on the optical transfer and the matrix, the 
reconstructed RGB signals may still sufier from a rest amount of colored aliasing 
according to the Nyquist-theorem. In order to still reduce the rest amount of 
aliasing, additionally to the luminance-reconstruction-fil- ter the color- 
reconstruction-filter may be applied. In particular therefore the color- 
reconstruction-filter comprises a false-color-filter to eliminate false colors from the 
input. 

[0052] As proposed by the invention and as outlined above already the 
contour-reconstruction-filter is applied in parallel with the color-reconstruction- 
filter. 

[0053] Preferably the false-color-filter is applied to a smallest possible array 
of green-pixels. Specifically the false-color-filter comprises steps in which the red- 
and blue-pixels in the smallest possible array of green-pixels are weighted by green- 
parameters and the weighted red and blue pixels are summarized by a median filter 
together with an average of one or more of the green-pixels in the array. An average 
of one or more green-pixels in the array may be talsen by appljdng an average filter. 
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In particular the average of green-pixels in a vertical direction and an average of 
green-pixels in a horizontal direction may be taken. The median-filtered pixels are 
compared with low-frequency-filtered pixels of the smallest possible array of green- 
pixels, whereby false colors are eliminated from the input. 

[0054] Such scheme advantageously allows for an adjustable false-color-filter, 
in particular the color-reconstruction-filter and/or the false-color-filter is applied to 
a smallest array of green-pixels having preferably an array-size of 3.times.3. Still, if 
necessary, such predetermined small array may also be of 5.times.5 size or larger. 
Such adjustment is independent of an optical low-pass-filter. Such array however 
should comprise at least four pixels, i.e. two green-pixels, one red-pixel and one 
blue-pixel. 

[0055] Preferably the applied color-reconstruction-filter has an array-size of 
3.times.3 or 5.times,5. In particular, an £irray-size of 5.times.5 is preferred in case of 
a heavy sensor matrix. 

[0056] Further, the green, and also the red and blue, color-reconstruction, i.e. 
the RGB-reconstruction, is fully independent of the 3.times.3 green false-color- 
array. The RGB-reconstruction-filter may be of 3.times.3 or 5.times.5 size. It is a 

particular advantage of the proposed method that the false-color functions are 
independent of the color-reconstruction-functions. 

[0057] Specifically the or a number of green-parameters may be used in the 
color-reconstruction-filter solely within the false-color-filter. 

[0058] Further a color-reconstruction-filter may be applied to an array size of 
3.times.3 or in particular to an array-size of 5.times.5 in case of a heavy sensor 
matrix. The coefficients of a corresponding filter function may be chosen as a 
function of the optical transfer and matrix. In particular this is outlined in a 
chapter 4 of the detailed description of a patent application with internal file 
number ID606638-I which has been filed on the same day as this application and 
which is incorporated by reference herein. Such methods maximize the resolution 
for near-white colors which for average scenes seem to be the most important eye 
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catchers. In particular sufficient resolution is provided while the amount for signal 
distortion is minimized. 

[0059] The rules for defining the filter coefficients of a 3.times.3 and a 
5.times.5 RGB color-reconstruction-filter are outlined for a preferred embodiment in 
particular in a patent application with internal file number ID505538-II which has 
been filed on the same day as this application and which is incorporated by 
reference herein. 

[0060] In a further preferred configuration a post-filter is applied in order to 
maintain in its output a phase to the output of a reconstruction-filter which has 
been applied previously. Advantageously the post-filter is applied subsequent to a 
false-color-filter, i.e. the post-filter is applied to the output of the false-color-filter 
after the RGB-reconstruction-filter. The reconstruction-filter may also be the 
luminance filter. Specifically the phase is fixed by advantageously applying 
subsequent to a false-color-filter a post-filter of 2.times.2-array-size by which a 
center-output-pixel of a smallest possible array of green-pixels is positioned in 
phase with a white-pixel. The white-pixel itself is centered with respect to the same 
array as to which a luminance-reconstruction-filter has been applied to. In 
particular signals are RGB-reconstructed on basis of a S.times.S or 5.times.5 array 
size and will get the same phase as the white compensated luminance signal due to 
the 2.times.2 post filter. It is in particular advantage of this configuration, that the 
false color-filter does not influence the phase. The post-filter advantageously also 
eliminates a green-non-uniformity of an image sensor. A preferred embodiment of 
the false-color-filter and particular ways of post-filtering may be taken from the 
above mentioned patent application with internal file number ID606638-II. 
[0061] Still also, as an alternative, a post filter may be left out for the use of a 
3.times.3 or 5.times.5 RGB color-reconstruction-filter in a further preferred 
embodiment, which is referred to as a smartgreen4-reconstruction-method, outlined 
in chapter 5 of the detailed description. This allows a simple and efficient 
processing and is therefore advantageous with regard to low-cost applications. 
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[0062] Further advantageously a low pass luminance signal is realized having 
a transfer characteristic as equal as possible to the total RGB transfer. The 
matching of a 4.times.4 or 6.times.6 low-pass-luminance-filter with respectively a 
3.times.3 or 5.times.5 green reconstruction filter is preferably achieved by a 
2.times.2 post-filter. 

[0063] In a further preferred configuration a high frequency luminance-filter 
is applied in dependence of an optical transfer of an optical system providing an 
image signal to the image sensor. Thereby a high-frequency aliasing free luminance 
component can be added to the low frequency reconstructed color signals in order to 
mask color artifacts. A corresponding preferred embodiment is described in the 
detailed description with reference to FIG. 2 of the drawing. 

[0064] The proposed method is adapted such that the optical transfer of the 
camera does not limit the application of a suitable filter. In other words, arbitrary 
optical low pass filters may advantageously be applied, merely depending on the 
application of a customer. It is even acceptable to apply no optical low pass filter at 
all. Especially this is achieved by the appliance of a false-color-filter as proposed, 
which is in particular adjustable independent of the optical transfer of the camera 
or of the sensor matrix. In particular, the false color filter can be adjusted as a 
function of the optical transfer of the camera at one half of the pixel frequency. 
[0065] Furthermore, various luminance-reconstruction-filters are offered for 
appliance. In particular a luminance-reconstruction-filter is applied to an array size 
of 2.times.2 in case of no or slight optical low pass filtering and further a respective 
luminance-reconstruction-filter is applied to an increased Eirray size of 4.times.4 or 
6.times.6 upon heavier low pass filtering. 

[0066] Furthermore, various color-reconstruction-filters are offered for 
appliance. In particular a 3.times.3 color-reconstruction-filter is applied in case of a 
2.times.2 luminance-reconstruction-filter or a 5.times.5 color-reconstruction-filter is 
applied in case of a 4.times.4-luminance-reconstruction-filter. 

[0067] Furthermore, various contour-reconstruction-filters are offered for 

appliance. In particular a 4.times.4-contour-reconstruction-filter is applied in case 
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of a 3.times.3-color-reconstruction-filter and further a 6.times.6-contour- 
reconstruction-filter is applied in case of a 5.times.5-color-reconstruction-filter. 
Advantageously the proposed method allows appliance of reconstruction-filters, 
which are adjustable as a function of the optical transfer of an optical system, e.g. of 
a camera or a sensor matrix. Therefore, depending on the chosen color- 
reconstruction-filter the output does hardly or not suffer from artifacts. A further 
advantage is, that the green-nonuniformity caused by a sensor is diminished. It is 
mostly a 5. times. 5 green reconstruction filter that suffers from green-green 
differences. In particular by the aid of a 3.times.3-green-reconstruction-filter in 
combination with the 2.times.2-post-filter such green-nonuniformity is removed or 
diminished. 

[0068] A preferred derivate of the proposed method is of particular interest 
and dedicated for low cost applications. It has been already mentioned above as 
smartgreen4 and combines the appliance of a 3.times.3-color-reconstruction-filter 
and a 5.times.5-contour-reconstruct- ion-filter. In particular subsequently a color- 
reconstructed and a contour-reconstructed signal are added for further processing. 
Smartgreen4 advantageously offers a low power consumption and a minimum of 
hardware. Further details are outlined in chapter 5 of the detailed description. 
[0069] As a particular preferred and advantageous configuration, the white 
compensated luminance reconstruction scheme of smartgreen as proposed above 
may be combined with a unique 5.times.5 aliasing and distortion free contour- 
reconstruction-filter. The contour-reconstruction-filter may be executed parallel to 
the color-reconstruction-filter. Such combination offers a strong reduction of 
artifacts in comparison with prior art filter schemes. IVIost advantageously such 
reconstruction scheme is based on the use of a 6.times.6 filter array. It may also 
incorporate several aliasing and distortion fi-ee parallel contour signals based on 
that array with the same performance as the above mentioned unique 5,times.5 
aliasing free contour-reconstruction-filter. 

[0070] In the whole processing chain as described above, i.e. the -RGB- 
reconstruction, in particulEir comprising the implementation of a post-filter for 
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phase matching; the luminance-signal processing; the color-reconstruction-signal 
processing, in particular comprising the implementation of a false-color-filter and in 
particular a further post-filter; and in parallel thereto the contour-signal processing, 
the amount of signal distortion is limited to an extremely low level. 
[0071] This also holds for a final JPEG-conversion. In a preferred 
configuration as well column- and also row-wise processing may be performed to 
execute the smartgreenS-reconstruction-algorithm according to the proposed 
method. Such processing advantageously reduces the amount of internal memory 
and the amount of data swapping to and from an external memory. This will 
support processing efifectivity and speed. Such measure also works if all data 
transfer is rotated by 90.degree. 

[0072] The proposed method is advantageously executed on an apparatus as 
proposed above, in particular on a computing system and/or a semiconductor device. 
Such system may advantageously comprise an intermediate memory interface 
located between an image sensor and a processing chip. Thereby advantageously 
the length and the number of rows of a pixel array to be filtered is no longer 
restricted, also the amount of data swap to and from an external memory of course 
should not delay the processing time too much. Consequently, still real-time 
processing is possible. The computing system may be any kind of processor unit or 
system or computer. 

[0073] Also real-time processing may preferably be performed without any 
memory using intermediate interface. 

[0074] However, in this case, for cost reasons, the total amount of available 
row delays may be limited, in particular to two. This may result in that only three 
vertical taps are available for the RGB reconstruction as well as for the realization 
for the contour signal. 

[0075] Further the invention leads to a computer program product storable on 
a medium readable by a computing system comprising a software code section 
which induces the computing system to execute the method as proposed when the 
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product is executed on a computing system, in particular when executed on the 
computing system of a camera. 

[0076] The invention will now be described with reference to the 
accompanying drawing. The detailed description will illustrate and described what 
is considered as a preferred embodiment of the invention. It should of course be 
understood that various modifications and changes in form or detail could readily be 
made without departing from the spirit of the invention. It is therefore intended 
that the invention may not be limited to the exact form and detail shown and 
described herein nor to anything less than the whole of the invention as disclosed 
herein and as claimed hereinafter. Further the features described in the description 
and the drawings and the clsdms disclosing the invention, may be essential for the 
invention considered alone or in combination. 

[0077] The detailed description accompanying the drawing provides the 
following chapters: 

[0078] 1. Processing flow of white compensated luminance reconstruction 

method 

[0079] 2. The white compensated luminance signal 

[0080] 3. RGB color reconstruction 

[0081] 4. Parallel ahasing free contour filtering 

[0082] 5. Smartgreen4. 

[0083] 6. Conclusions 

[0084] 1. Processing Flow of the White Compensated Luminance 

Reconstruction Method 

[0085] In FIG. 1, a part of a general architecture of an integrated circuit with 

a central bus and an external memory interface is shown. A sensor signal is offered 
via the central bus to the external memory. For the realization of the RGB-color- 
reconstruction and of the parallel contour-reconstruction the sensor data is 
retrieved from the external memory via the central bus to the reconstruction. After 
reconstruction the data are sent directly to the processing blocli or sent back to the 
external memory. 
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[0086] The processing block contains more or less standardized camera 
functions as the matrix, white balance, knee and gamma. Sending the reconstructed 
data directly to the processing is an important issue because in order to obtain a 
fast execution time for still pictures or for video data, the amount of time consuming 
data swap from and to the memory should be limited. 

[0087] In FIG. 2 a more detailed block diagram of a preferred embodiment of 
the proposed method, hereinafter referred to as "smartgreen3"-signal- 
reconstruction, is shown. Via the central bus the sensor data is sent from the 
external memory in small packets to a small internal memory array in the 
reconstruction block. From this [lS.times.64H.times.6V] array in FIG. 2, containing 
1 Signal (16 bits) and 64 Horizontal by 6 Vertical pixels (768 bytes) as an example, 
the raw sensor data can be randomly retrieved for reconstruction. In particular, a 
row of sensor data can be processed in multiples of (64-2*ho) pixels, where ho is the 
horizontal offset of the filter array. For an n.times.m reconstruction array, where 
"n" are horizontal and "m" are vertical pixels, the offset is ho=n div 2 "div" means 
rounding in the direction of zero to the nearest integer. Therefore, it holds ho=l for 
n=3, ho=2 for n=4 or n=5 and ho=3 for n=6. The first pixel that can be reconstructed 
is at position 1+ho, the last one at position N-ho, where N is the total number of 
pixels in a sensor row. Reconstructing a complete sensor row requires N/(64-2*ho) 
packages to be sent to the reconstruction block. In the lower part of FIG. 2, the RGB 
color signals are reconstructed with the raw sensor data. The (low) frequency 
transfer, by means of the choice of the filter weights, depends on the optical transfer 
of the camera. Whether the false color killer and the 2.times.2 post-filters are used 
or not also depends on the optical transfer. In the upper part of FIG. 2 the R and B 
pixels are multiplied with the smartgreen parameters. The smartgreen parameters 
may e.g. be retrieved according to the method disclosed in WO 99/04555 or in EP 01 
200 422.2. With this specific signal three aliasing free luminance signals are 
reconstructed: the contour signal, the white compensated luminance signal Yn and 
a low frequency luminance signal Ylf The latter has about the same transfer 
characteristic as the reconstructed RGB signals. By subtracting the low frequency 
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signal Ylf from the white compensated luminance signal Yn, a high frequency 
luminance component is generated: (Yn-Ylf). It is important to be aware of the fact 
that in the preferred embodiment, in order to prevent undesired false colors at 
higher frequencies, the (Yn-Ylf) signal if ever possible should not earlier be added to 
each color signal then after the matrix and white balance functions in the 
processing block. The very same counts for the contour signal. Preventing the 
mentioned undesired false colors is the reason why the total output of the 
reconstruction block consists of four or respectively three signals. Besides the 
fastest possible execution time this is a second reason why it is preferred to send all 
signals directly to the processing block. A third reason can be that it makes twice a 
[4S,times.64H.times.lV] internal memory superfluous, one in the reconstruction 
block for sending the four signals to the external memory and one in the processing 
block for retrieving them again. The [4S.times.64H.times.lV] internal memory 
stands for the storage of four (16 bit) signals of (64-2*ho) horizontal pixels of one 
vertical row, containing a total of 640 bytes when rounding to 64 pixels. 
[0088] When maximum flexibility of the design is required, e.g. for a time 
consuming reconstruction and/or processing by means of specific software via the 
CPU, then the two [4S.times.64H.times.lV] internal memories, one for the 
reconstruction and one for the processing, should be applied. 

[0089] At the top of FIG. 2 the aliasing free contour is realized followed by the 
overshoot control processor which prevents over- and undershoots at lower 
frequencies. Two-dimensional sharpness improvement using a two-dimensional step 
transient signal is achieved by the realization of a two-dimensional detection signal 
which is suited for controlling overshoots. This so called step transient signal can be 
used for several overshoot (and undershoot) control methods, and results in a very 
attractive sharpness improvement without exaggerated and unnatural looking 
overshoots. It is allowed to add the contour and the (Yn-Ylf) signal to a single signal 
that is sent to the internal [4S.times.64H.times.lV] memory. 
[0090] The following chapters may be summarized as follows. 
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[0091] In chapter 2, the reaUzation of an ahasing free and zero distortion 
luminance signal Yn with a 2.times.2 and 4.times.4 filter array is described. Details 
of a luminance-reconstruction-filter are disclosed in particular in chapter 2 of the 
detailed description of a patent application with internal file number ID606638-I 
which has been filed on the same day as this application and which is incorporated 
by reference herein. Also low frequency luminance signals with a 4.times.4 and a 
6.times.6 low pass filter eirray are explained in chapter 3 therein. The realization of 
the low frequency RGB reconstruction with or without a false color killer, is further 
described. Details of a color-reconstruction-filter are disclosed in particular in 
chapter 3 of the detailed description in a patent application with internal file 
number ID606638-II which has been filed on the same day as this application and 
which is incorporated by reference herein. The realization of a 5.times.5 aliasing 
free contour-filter is described in chapter 2 and 4.times.4 and 6.times.6 ediasing free 
contour signals are described in chapter 4 in the following. 

[0092] The concept proposed in this application and the filters described 
herein may also be adapted in a flexible way as a function of the optical transfer 
and sensor matrix. This is also described in detail in chapter 4 of the detailed 
description of above patent application with internal file number ID606638-I. 
[0093] 2. The White Compensated Luminance Signal 

[0094] The white compensated luminance signal is based on the calculation of 
the sm£irtgreen parEimeters. Examples are shown in WO 99/04555 and EP 01 200 
422.2. The SmartGcntrlR and SmartG^ntrlB parameters are referred to as: wbr and 
wbb respectively. FIG. 3 shows that the red and blue pixels are multiplied with the 
smartgreen parameters. Next the four pixels are added together resulting in a 
white compensated luminance pixel Yn. The center of pixel Yn is shifted half a pixel 
to the right and to the bottom, considering the R pixel as the first one. The 
consequence is that all other signals that have to be reconstructed, i.e. red-green- 
blue and contour, should get the same center position as this Yn signal. 
[0095] The advantages of the white compensated luminance signal Yn are the 
very same as for the 5.times.5 aliasing free contour signal. An aliasing free contour- 
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signal for RGB Bayer image sensors is based on a unique 5,times.5 parallel contour 
filter that, without the need for an optical low pass filter, has a zero throughput at 
the first RGB sample frequency. Its signal distortion is almost zero, resulting in a 
contour signal without visible artifacts. The so far known contour filters amplify the 
back folded and undesired frequencies outside the Nyquist domain of the sensor as 
well. This will lead to distortions and as a consequence a better visibility of the 
unwanted ahasing components in the picture. This unique 5.times.5 contour filter 
prevents those aliasing artifacts and moreover eliminates the green-green 
differences caused in the green channel of the image sensor. Particular advantages 
are: 

[0096] 1. Without the need of an OLPF (optical low pass filter), the Yn signal 
has a zero throughput at the first RGB sample fi-equencies, so it will cause no 
aliasing at those points. At the second and higher multiples of the sample 
frequencies the throughput is low, but the low pass of the lens and the modulation 
transfer function (MTF) of the sensor will be effective there as well. 
[0097] 2. The signal distortion is almost zero, resulting in a luminance signal 
Yn without visible airtifacts. 

[0098] In the following it will be explained why a 2.times.2 white 
compensated luminance filter is particularly advantageous. 

[0099] 1. The 2.times.2 unity is needed for the elimination of the modulated 

color information in the sequential RGB Bayer color signal of the image sensor and 
the elimination of the green non-uniformity caused by the image sensor. Preventing 
green non-uniformity in the parallel contour signal of RGB Bayer image sensors 
and is achieved by a method for developing a two-dimensional parallel contour filter 
that eliminates the green-green differences caused by the image sensor. Parallel 
contour processing for Bayer image sensors allows to use the green signal from the 
image sensor for generating a two-dimensional contour signal in parallel with the 
RGB color reconstruction. 

[00100] The advantage is that no extra row delays are needed as it is the case 
with serial contour processing. It has to be noted that whatever OLPF-type has 
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been used, it does neither influence this color modulation in large saturated color 
areas nor the green non-uniformity. 

[00101] 2. The multiplication by the smartgreen parameters is needed in order 
to undo the effect of a matrix, wherein the sum of the coefficients of the matrix is 
not equal to unity, and/or the effect of the environmental color temperature. 
[00102] These are the two reasons why the smartgreen parameters are 
calculated. If those parameters are not unity, the color sensor will not act as a black 
and white sensor. 

[00103] A slight disadvantage of the 2.times.2 white compensated luminance 
signal Yn may be that the resolution will be reduced when an optical low pass filter 
(OLPF) is applied. However this can be compensated by applying 4.times.4 filters 
for the reconstruction of Yn. Depending on the amount of the low pass transfer of 
the OLPF, a Yn filter with heavier weights can be chosen. On the left hand side of 
FIG. 6 the already described 2.times.2 Yn filter is shown. In the middle and on the 
right hand side two 4.times.4 Yn filters are shown. The right one is the heaviest' 
filter. Besides for compensating the transfer loss of an OLPF they also can be 
applied for compensation of the transfer loss of the lens. 

[00104] The 4.times.4 Yn filters restore the green uniformity and eliminate the 
modulation in colored areas as well. The green uniformity restoration can be 
achieved by preventing green non-uniformity in the parallel contour signal of RGB 
Bayer image sensors which may be performed by developing a two-dimensional 
parallel contour filter that eliminates the green-green differences caused by the 
image sensor. Parallel contour processing for Bayer image sensors allows to use the 
green signal from the image sensor for generating a two-dimensional contour signal 
in parallel with the RGB color reconstruction. The advantage is that no extra row 
delays are needed as is the case with serial contour processing. As a rule one may 
say: The subtraction of neighbor diagonal filter coefficients should result in a zero 
contribution. This will average and, as a consequence, eliminate the green-green 
differences of the neighbor green pixels. 
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[00105] The combinations of the negative coefficients always result in an 
addition of two green and a red a blue pixel, and also have the advantage of a zero 
transfer for the color modulation. It is to be noted that depending on the 
combination of the total optical transfer and the heaviness of an arbitrary 4.times.4 
Yn filter, finally undesired over- and undershoots can occur. The choice of the 
coefficients can only be done when the optical transfer is known, for example by 
shooting and processing a zone plate scene. 
[00106] 3. RGB Color Reconstruction. 

[00107] The reconstruction method described in this preferred embodiment 
should be suited for cameras with an £irbitrary optical low pass filter (OLPF) or 
with no OLPF at all. The choice of an appropriate OLPF is not a simple matter. In 
most cases it is a compromise between the loss of sharpness and the acceptance of a 
certain amount of aliasing or preferably no aliasing at all. Relevant parameters are: 
[00108] 1. the optical transfer of the lens; 

[00109] 2. the modulation transfer function (MTF) of the sensor, including a 
possible pixel microlens, determining light sensitive pareimeters; 
[00110] 3. the sensor matrix, determined by the color array of the sensor. An 
extra amount of aliasing may be caused by the heaviness of the matrix and 
influence the specification of the OLPF transfer. 

[00111] 4. a given reconstruction algorithm, which of course results in a certain 
transfer characteristic as well, causing the need for more or less aliasing reduction 
by means of the OLPF. 

[00112] If one of those four parameters in a camera has been modified, one 
should consider the adaptation of an OLPF. Here a problem arises which is that 
usually an OLPF is specified without taking all four mentioned parameters into 
account. Therefore, in the preferred embodiment it is tried to anticipate on an 
existing or a preferred camera design, i.e. the optics (pt. 1) and the sensor (pt. 2 and 
3), a reconstruction method that offers some flexibility (pf 4). The transfer of the 
reconstruction filters can be changed by choosing certain filter weights, or the 
possibility to apply a false color killer. 
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[00113] In chapter 5 a particular preferred RGB color reconstruction filter is 
explained: using a 3.times.3 array. Also a 5.times,5 array may be used for a RGB 
color reconstruction filter as described in a chapter 3.6 of the detailed description of 
a patent application with internal file number ID606638-II, which has been filed on 
the same day as this application £uid which is incorporated by reference herein. 
Both can be applied with (or without) a false color killer. 
[00114] 4. Parallel Aliasing Free Contour Filtering 

[00115] Similar to the unique 5.times.5 aliasing fi-ee contour filter mentioned 
above, the 4.times.4 and 6.times.6 aliasing free contour filters have a zero 
throughput at the first RGB sample frequency independent of the optical low pass 
transfer. Also the signal distortion is almost zero, as for the (Yn-Ylf) signal. This 
means that the aliasing free contour signal has no visible artifacts, because there is 
no back folding of undesired frequencies outside the Nyquist domain of the image 
sensor. Until now only one unique 5.times.5 aliasing free contour filter exists, 
however there are many 4.times.4 and 6.times.6 ahasing free contour filters 
possible, which all eliminate the green non-uniformity as well. In FIG. 5 three 
4.times.4 and in FIG. 6 three 6.times.6 contour filters are shown. From the left to 
the right the 4.times.4 contour filters have an increasing throughput for high 
frequencies. In FIG. 8 from the top to the bottom the three transfer characteristics 
of the 4.times.4 filters of FIG. 4 respectively are shown. In FIG. 9 the three different 
transfer characteristics of the 6.times.6 filters of FIG. 6 respectively are shown 
starting at the top with the left 6.times.6 filter of FIG. 6. The middle contour filter 
of FIG. 6 has less amplification at the higher frequencies as compared to the left 
and the right one. This can clearly be seen in FIG. 9. 

[00116] By leaving out the upper and bottom row of the 6.times.6 filters in 
FIG. 6, 6.times.4 filters are obtained with a sigmaContour of respectively 68, 52 and 
84. The reason this is mentioned is because it offers the possibility of a 
smartgreen3-method with 4 rows or respectively 6 rows. On purpose the first 
4.times.4 filter of FIG. 5 has the very same weights as a first (Yn-Ylf) filter, just as 
the first 6.times.6 filter of FIG. 46 is the same as a further (Yn-Ylf) filter. Again 



-23- 



CIPM-PHI-CA0669 



there is a possibility to combine the filter weights of the (Yn-Ylf) and the contour 
filter, resulting in a fiirther simplification of the smartgreen3-method design. If 
done so, then the coring or a local noise reduction (LNR) or an overshoot control 
processing (OCP) may be performed on the total obtained contour signal, or on a 
part of it, as it is shown in FIG. 7. The dashed part illustrates that a part of the 
contour signal, without LNR and OCP, can be applied as a (Yn-Ylf) signal. From a 
performance point of view there are no objections to do so. It is still possible to 
switch off the contour signal for demonstration purposes and, if switched on, to 
control its parameters. This possibility has same advantages with regard to a 
variety of applications. The only disadvantage of the block diagram of FIG. 7 is that 
trying out another contour filter transfer, has consequences for the (Yn-Ylf) signal 
as well. Therefore, if a convenient adjustment and having some flexibility, is given 
priority it may be preferred in an alternative not to combine the (Yn-Ylf) filter and 
the contour filter. 

[00117] As already stated, the choice of the (Yn-Ylf) filter is not too critical. 
Therefore, it does not make sense to describe how to combine its filter weights with 
the weights of the contour filter in order to obtain a single filter. 
[00118] The choice of the contour filter type and the adjustment of the amount 
of contour depends on the personal taste at judging several pictures. The only 
directive that can be given is that it is rather logical to apply a 4.times.4 contour 
filter when a 3.times.3 color reconstruction has been chosen and a 6.times.6 contour 
filter in case of a 5.times.5 color reconstruction. Further, while adding contour, 
special attention should be paid to the desaturation of colored edges. 
[00119] In the following, two very sophisticated contour processing features as 
overshoot control processing (OCP) and local noise reduction (LNR) is explained. A 
general procedure for OCP may use a continuous luminance signal as it is available 
for example in TV-applications. In such a procedure, two dimensional sharpness 
improvement using a two-dimensional step transient signal is achieved by the 
realization of a two-dimensional detection signal that is suited for controlling 
overshoots. This so called step transient signal can be used for several overshoot, 
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and also undershoot, control methods, resulting in a very attractive sharpness 
improvement without exaggerated and unnatural looking overshoots. In 
multiplexed RG/RB sensors special attention should be paid to the most important 
goal of the smartgreenS processing, being the prevention of visible artifacts. 
Regarding the LNR, a method for a very specific two-dimensional contour filter can 
be applied. By such a method a contour with local noise reduction for digital 
electronic cameras is achieved by the realization of a very specific two-dimensional 
contour filter that is capable to reduce the amount of noise as well as the sharpness 
by means of a two-dimensional standard deviation function. An arbitrary contour 
filter, as applied with sm£irtgreen3 and described above, preventing visible artifacts 
is £ilso possible. 

[00120] In this chapter a smartgreenS derivate will be described which is 
referred to as smartgreen4. It consists of a 3.times.3 color reconstruction, does not 
use the (Yn-Ylf) signal and should preferably only be applied with two possible 
contour filters, a 5.times.5 or a 5.times.3 one. It is particular suitable for low cost 
applications. 

[00121] 5. Smartgreen4 

[00122] A very fast processing method with a very acceptable result, better 
than the smartgreenl/2 methods, is offered by the combination of, on the one hand, 
the 3.times.3 RGB reconstruction as described in a chapter 3.1 of the detailed 
description and, on the other hand, the 5.times.5 aliasing free contour filter as 
mentioned above. 

[00123] The mentioned unique 5.times.5 parallel contour filter has without the 
need for an optical low pass filter, a zero throughput at the first RGB sample 
fi-equency. Its signal distortion is almost zero, resulting in a contour signal without 
visible artifacts. The so far known contour filters amplify the back folded and 
undesired fi-equencies outside the Nyquist domain of the sensor as well. This will 
lead to distortions and as a consequence a better visibility of the unwanted aliasing 
components in the picture. This unique 5.times.5 contour filter prevents those 
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aliasing artifacts and moreover eliminates the green-green-difFerences caused in the 
green channel of the image sensor. 

[00124] The transfer characteristics of the RGB color-reconstruction-filter- s 
are shown in FIGS. 10 and 11 and have better matching characteristics for a 
3.times.3 green-reconstruction-filter as conventional filters. In the upper-left comer 
the filter weights are displayed. 

[00125] Although a 2.times.2 post-filter will eliminate green-green differences, 
also this green-reconstruction-filter will already do that. A method for developing a 
two-dimensional parallel contour filter that eliminates the green-green differences 
caused by the image sensor can be applied. 

[00126] As a rule, "the subtraction of neighbor diagonal filter coefiicients 
should result in a zero contribution. This will average and as a consequence 

eliminate the green-green differences of the green-pixels". The use of this rule has 
also been explained in the description of the white compensated luminance signal in 
chapter 2. 

[00127] The advantages of better matching green filters are that: 

[00128] they will cause less greenish interferences at higher diagonal 

frequencies 

[00129] less false color killer artifacts will occur, 
[00130] the signal is free of artifacts at edges. 

[00131] It is to be noticed that a difference with the smartgreenl/2 
reconstruction method is that smartgreen4 like smartgreen3 does not apply the 
smartgreen signal as a color signal. The smartgreen signal is only used for the false 
color detector as illustrated in FIG. 13. The lack of artifacts in lower image parts is 
mainly caused by green transfer characteristics. The filters described here are 
matching better than the filters in conventional reconstruction methods. This is a 
particular advantageous difference, which offers a surprisingly perfect 
reconstruction at colored and black and white edges. This makes the green filter 
combination of the FIGS. 10 £ind 11 very unique and it will be explained in the 
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following how the 3.times.3 reconstruction-filter is applied within the smartgreen4- 
reconstruction. 

[00132] The method is referred to as smartgreen4 and it offers all the 
mentioned advantages of both, reconstruction methods and also methods for the 
elimination of green-green differences. Because it is fast it can be applied for video 
signal reconstruction or, depending on the application, for still images as well. It is 
an interesting option to implement the possibility of this unique combination in a 
smartgreen3 chip design. Also for dedicated applications, requiring low power and a 
minimum of hardware, as for example with CMOS sensors, this combination seems 
to be a very advantageous embodiment. For dedicated high volume and low cost 
CMOS applications again fixed row delays can be applied in order to avoid external 
memory. In FIG. 12 the basic block diagram for such an application is shown with 
an internal memory of [lS.times.row.times.4V], i.e. four dedicated row delays. In 
FIG. 13 a more detailed diagram of the reconstruction is shown. 
[00133] The false color detector acts with a smartgreenl signal and applies 
respectively for the presence of green and for the absence of green. In case of the 
absence of green the detector should only be applied using a 5.times.5 array. 
Therefore here it has been chosen for the simplest version, using the smartgreenl 
signal for the center row only. Four row delayed signals are offered to the 5.times.5 
aliasing free contour reconstruction filter. The actual sensor signal contains the 
data of row 1. No local noise reduction, or overshoot control has been applied, 
however this is of course possible. Only a simple coring circuit reduces the noise. 
[00134] At the left-top of FIG. 14 the result of the above described processing is 
shown. For all four quadrants of the artificial zone plate the same optical low pass 
filter has been applied as well as the matrix of the Philips FT19 sensor. 
[00135] At the top-right the 5.times.5 aliasing free contour signal of the zone 
plate is shown (with a gain of 0.7). As can be seen it is completely free of 
interferences. 

[00136] At the bottom-right the 3.times.3 RGB reconstruction by use of filters 
of FIGS. 10 and 11 is shown combined with a very specific 5.times.3 parallel contour 
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filter, which advantageously eliminates the green-green dififerences caused by the 
image sensor. Also for a further price reduction such a filter is very interesting 
because then only two instead of four row delays are needed. As can be seen the 
performance is less good. The sharpness has become asymmetrical due to the loss in 
the vertical direction and there are more interferences. Further more greenish 
aliasing occurs around the RGB sample frequency in the outer bottom-left comer. 
At the bottom-right only the 5.times.3 contour signal is shown (gain 1.0) which can 
be easily compared with the above lying 5.times.5 aliasing free contour. Most 
preferably, the contour signals also have been added to the RGB signals after the 
matrix and white beilance processing. 

[00137] For the processed results on the left hand side the above described 
false color killer has been applied. 

[00138] Both applications shown in FIG. 14, at the top with the 5.times.5 
aliasing free contour and at the bottom with the specific 5.times.3 contour, should 
be judged as smartgreen4 processing methods. 
[00139] 6. Conclusions 

[00140] The following particular advantages are achieved with the 
smartgreenS reconstruction-method compared with smartgreenl and smartgreen2: 
[00141] It is a flexible design. Depending on the heaviness of the sensor matrix 
a choice can be made between two color reconstruction filters. As a consequence the 
(low pass) luminance filters c£in be defined application specific. Several high 
frequency luminance filters can be chosen and adjusted as a function of the optical 
transfer of the camera. 

[00142] If the false color killer is needed, then depending on the chosen color 
reconstruction, it does hardly or not suffer from visible artifacts (black and white 
dots). The reason is that a stronger low pass filtered color reconstruction is used 
than with smartgreenl or smartgreen2. 

[00143] There are no artifacts at all due to the smartgreen reconstruction for 
the simple reason that the smartgreen concept is used only for the false color 
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detection and not as a color signal. In order to compensate the loss of resolution, a 
high frequency luminance signal is generated. 

[00144] Depending on the chosen color reconstruction, the bandwidth of the 
back folding colored aliasing is smaller than with smartgreenl and smartgreen2. 
[00145] The transfers of the red, green and blue reconstruction filters are 
matched in such a way that less or no greenish interferences will occur at higher 
diagonal frequencies as it is the case with the smartgreenl and smartgreen2 filters. 
[00146] The color reconstruction filters are fi*ee of post stamp border looking 
artifacts at color and luminance edges. 

[00147] All generated luminance signals are free of aliasing and distortion by 
means of the so called luminance white compensation in combination with the filter 
weights. 

[00148] SmartgreenS offers several aUasing free contour filters which can be 
chosen as a function of the optical transfer of the camera. 

[00149] All filters of the smartgreenS design, presented here, eliminate the 
green non-uniformity caused by the sensor. The 5.times.5 green reconstruction filter 
finally does it with the help of the 2.times.2 post-filter. 

[00150] In summary, due to its flexibility, the lack of signal distortion and its 
false color killer, smartgreenS is well suited for the reconstruction and processing of 
digital still images and of contiguous video pixels of sensors having a Bayer color 
filter array. 

[00151] An interesting derivative of smartgreenS is smartgreen4. By offering a 
low power consumption and a minimum of hsirdware smartgreen4 is suited for 
dedicated still and video applications. 



